One common source of image blur in digital imaging is the relative motion between a camera and a scene during the exposure time (integration time) associated with image capture. This type of image blur is sometimes called “motion blur” or “smear.” Motion blur typically occurs when the light level in the image capture environment is dim, thus necessitating a long exposure time.
One approach to reduce motion blur is to use an electronic flash to supplement the natural illumination. However, this is only effective when the subject is relatively close to the camera. Additionally, many users find that flash photography provides a less satisfactory experience.
Another approach to reducing the problem of motion blur is to improve the light sensitivity of electronic image sensors in order to reduce the exposure time during image capture. While much progress has been made in this area in recent years, the light sensitivity of currently available electronic image sensors is not high enough to prevent motion blur in many image capture environments.
One factor that reduces the light sensitivity of electronic image sensors is the use of color filters placed over the sensors to enable the formation of color images. For example, the well-known “Bayer pattern” taught in U.S. Pat. No. 3,971,065 to Bayer, teaches the use of a repeating array of red, green and blue color filters to detect color image signals. While this general approach is used today in most consumer digital cameras, the color filter array (CFA) has the undesirable effect of throwing away about ⅔ of the incident light, and therefore substantially reduces the photographic speed of the imaging system.
U.S. Pat. No. 4,876,591 to Muramatsu discloses an electronic imaging system that includes a beam-splitter and two different sensors, wherein one sensor has no color filters and the other sensor includes a pattern of color filters. The sensor without the color filters provides for increased light sensitivity, while the other sensor provides color information. Although this system improves the light sensitivity over a single conventional image sensor, the overall complexity, size, and cost of the system is greater due to the need for two sensors and a beam splitter. Furthermore, the beam splitter directs only half the light from the image to each sensor, limiting the improvement in photographic speed.
U.S. Patent Application Publication No. 2007/0046807 to Hamilton, et al., teaches a digital image system using a single sensor having some color image pixels with color filters and some panchromatic image pixels having no color filters. An interpolation algorithm is used to reconstruct a full-color image where the higher speed panchromatic image pixels provide the image detail information. While this approach can reduce motion blur to some extent by enabling shorter exposure times, there will still be some level of motion blur in many low-light imaging scenarios.
Another method to reduce the affect of motion blur in digital images is to use an image enhancement algorithm to compensate for blur in the captured image. Such algorithms are often referred to as “deblurring” or “deconvolution” algorithms. Such algorithms can be roughly classified into two categories: “blind” and “non-blind”. If a blur kernel associated with the image blur is not known, then the problem is said to be “blind deconvolution,” whereas when the blur kernel is known, it is said to be “non-blind deconvolution.”
For non-blind deconvolution, the most common technique is Richardson-Lucy (RL) deconvolution. (See the articles: W. H. Richardson, “Bayesian-based iterative method of image restoration,” Journal of the Optical Society of America, Vol. 62, pp. 55-59, 1972, and L. B. Lucy, “An iterative technique for the rectification of observed distributions,” Astronomical Journal, vol. 79, pp. 745-754, 1974.) This method involves the determination of a deblurred image (sometimes referred to as a “latent image”) under the assumption that its pixel intensities conform to a Poisson distribution.
In the article “Improved image deblurring with anti-reflective boundary conditions and re-blurring” (Inverse Problems, Vol. 22, pp. 2035-2053, 2006), Donatelli et al. use a Partial Differential Equation (PDE)-based model to recover a deblurred image with reduced ringing by incorporating an anti-reflective boundary condition and a re-blurring step.
In the article “Progressive inter-scale and intra-scale non-blind image deconvolution” (ACM Transactions on Graphics, Vol. 27, Iss. 3, 2008), Yuan, et al. disclose a progressive inter-scale and intra-scale non-blind image deconvolution approach that significantly reduces ringing.
Blind deconvolution is an ill-posed problem, which is significantly more challenging. Approaches to pure blind deconvolution apply to either single blurred image or multiple blurred images. The most challenging problem is single-image blind deblurring, which requires the simultaneous estimation of the deblurred image and the Point Spread Function (PSF) associated with the image blur.
In the article “Removing camera shake from a single photograph” (ACM Transactions on Graphics, Vol. 25, pp. 787-794, 2006), Fergus et al. show that blur kernels are often complex and sharp. They teach an ensemble learning approach to recover a blur kernel, while assuming a certain statistical distribution for natural image gradients.
In the article “High-quality motion deblurring from a single image” (ACM Transactions on Graphics, Vol. 27, pp. 1-10, 2008), Shan et al. disclose a method for removing motion blur using a unified probabilistic model of both blur kernel estimation and unblurred image restoration.
In the article “Understanding and evaluating blind deconvolution algorithms” (Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 2009), Levin, et al. described and evaluated a number of single-image blind deconvolution algorithms.
Having multiple blurred images can provide additional constraints to improve the deblurring process. In the article “Two motion-blurred images are better than one” (Pattern Recognition Letters, Vol. 36, pp. 211-217, 2005), Ray-Acha, et al. teach the use of images with different blurring directions to provide improved kernel estimation.
Recently, another type of blind deconvolution has been disclosed that employs additional information besides the blurred image to improve the deconvolution. This method can be categorized as “quasi-blind deconvolution.” In the article “Simultaneous image formation and motion blur restoration via multiple capture” (Proc. International Conference Acoustics, Speech, Signal Processing, pp. 1841-1844, 2001), Liu, et al. teach using a CMOS sensor to capture multiple high-speed frames within a normal exposure time. Image pixels having motion blur are replaced with the pixels from one of the high-speed frames.
In the article “Motion deblurring using hybrid imaging” (Proc.
IEEE Conf on Computer Vision and Pattern Recognition, Vol. 1, pp. 657-664, 2003), Ben-Ezra, et al. disclose a hybrid camera that simultaneously captures a high-resolution image together with a sequence of low-resolution images that are temporally synchronized. With this method, optical flow is derived from the low-resolution images to estimate the global motion blur of the high-resolution image.
In the article “Coded exposure photography: motion deblurring using fluttered shutter” (ACM Transactions on Graphics, Vol. 25, pp. 795-804, 2006), Rasker, et al. disclose a “fluttered shutter” camera, which opens and closes the shutter during a normal exposure time with a pseudo-random sequence. The flutter changes the normal “box filter” to a broad-band filter that preserves high-frequency spatial details in the blurred image. As a result, the corresponding deconvolution becomes a well-posed problem.
In the paper “Image deblurring with blurred/noisy image pairs” (ACM Transactions on Graphics, Vol. 26, Issue 3, 2007), Yuan et al. have disclosed a method of image deblurring using blurred and noisy image pairs. Each image pair contains a blurred image captured with a long exposure time, and a noisy image captured with a short exposure time. The noise associated with the short exposure time image can be severe under low light condition, and therefore the deblurring results are highly depend on the performance of a denoising operation.
In the article “Image and depth from a conventional camera with a coded aperture” (ACM Transactions on Graphics, Vol. 26, Issue 6, 2007), Levin et al. employ a coded to obtain an approximate blur kernel which can be used in a deblurring algorithm. This deblurring approach is limited to image blur caused by defocus.
Another method to reduce blur and capture images in low-light scenarios, known in the fields of astrophotography and remote sensing, is to capture two images: a panchromatic image with high spatial resolution and a multi-spectral image with low spatial resolution. The images are fused to generate a multi-spectral image with high spatial resolution. Such approaches are described in U.S. Pat. Nos. 7,340,099, 6,011,875 and 6,097,835.